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PREFACE 

Experience  indicates  that  the  major  sources  of  uncertainty  in 
weapon  systems  development  can  be  traced  to  those  activities  which 
involve  testing  and  redesign.  Yet,  surprisingly  little  of  a  con¬ 
ceptual  nature  has  been  dons  to  improve  the  decisionmaking  process 
involved  in  performing  these  activities.  In  this  paper,  an  adaptive 
model  of  the  development  process  is  presented  which  enables  decision¬ 
makers  to  determine  which  tests,  if  any,  should  be  performed  at  a 
given  stage  and  what  corrective  action,  if  any,  should  be  taken  once 
the  test  results  are  known. 

This  study  should  be  useful  to  the  System  Project  offices,  the 
Air  Force  Research  and  Technology  Division,  the  Air  Staff,  and  the 
directors  of  various  test  facilities.  It  should  also  be  of  Interest 
to  the  Office  of  the  Secretary  of  Defense  and  to  defense  contractors, 
as  well  as  to  persons  concerned  with  cotranercial  development. 
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S'JMMARY 


This  Memorandum  considers  the  problem  of  selecting  "optimal"  or 
"best"  programs  for  the  conduct  of  testing  and  redesign  activities 
in  the  development  of  military  hardware.  An  adaptive  model  of  the 
testing  process  Is  presented  which  is  designed  to  (1)  provide  the 
project  director  and  hls  staff  with  a  meant  for  determining  the  beat 
cesr  to  perform  at  a  given  stage  In  the  development  of  a  system,  and 
(2)  enable  the  same  decisionmakers  to  choose  Intelligently  between 
the  available  redesign  actions  once  the  teat  results  are  known.  While 
the  model  la  presented  In  terms  of  relatively  simple  systems  and  tea;*. 
It  Is  capable  of  handling  systems  and  teats  of  a  highly  complex  nature. 

The  final  checkout  phase  of  a  new  system  is  considered  first. 

The  model  developed  by  Johnson  [5]  for  the  purpose  of  determining  the 
optimal  trouble-shooting  procedure  when  an  operating  system  falls  is 
found  to  be  an  appropriate  abstraction  of  this  phase  and  is  adopted 
as  a  starting  point.  This  model  Is  then  generalised  to  Include  the 
case  In  which  the  object  tested  Is  destroyed  when  the  test  Is  unsuccess¬ 
ful.  In  addition,  sufficiency  conditions  for  the  optimality  of  the 
test  procedure  suggested  by  Johnson  are  given. 

In  Sec.  Ill  the  problem  of  test  selection  In  the  early  stages  of 
development  Is  superimposed  on  the  final  checkout  phsse.  It  Is  well 
known  that  the  Imperfect  nature  of  the  Information  gained  In  early 
testing  tends  to  offset  the  advantages  of  early  redesign.  A  dynamic 
progr aiming  formulation  is  given  which  enables  one  to  determine  which 
test,  If  any,  should  be  performed  at  a  given  stage  and  what  corrective 
action.  If  any,  should  be  taken  once  the  test  result  Is  known.  As  a 
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by-product,  a  means  for  determining  the  net  benefit  of  a  given  test 
at  a  given  stage  is  obtained.  The  use  of  the  multistage  decision 
model  developed  in  th  .<  .ection  is  illustrated  by  means  of  a  compre¬ 
hensive  example. 

The  situation  ,.t.  .rich  there  is  serial  dependence  among  certain 
test  information  messages,  as  in  the  case  of  a  multistage  rocket,  is 
examined  in  Sec.  IV.  Zn  the  last  section,  it  is  shown  that  Nelson's 
model  of  parallel  R&D  efforts  [6]  may  be  viewed  as  a  special  case  of 
the  model  developed  in  this  paper. 


t 
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I.  INTRODUCTION 


The  aim  of  this  study  is  to  establish  a  framework  and  a  methodol¬ 
ogy  for  selecting  ''optimal"  or  "best"  urograms  for  the  conduct  of  tests 
in  the  development  of  military  hardware.  For  expository  purposes,  the 
end  product  of  the  development  process  may  be  considered  to  be  a  new 
weapon  system  of  some  complexity,  such  as  a  missile,  a  fighter,  or  a 
radar  system.  The  hardware  to  be  developed,  which  will  be  called  a 
system,  will  be  viewed  as  consisting  of  several  subsystems  (e.g. ,  pro¬ 
pulsion,  guidance).  Each  subsystem,  in  turn,  is  considered  to  be  made 
up  of  several  components ;  each  component  may  consist  of  parts .  etc. 

Systems  whose  subsystems  are  indivisible  will  be  referred  to  as  one- 
level  systems;  systems  whose  subsystems  are  at  most  made  up  of  indivis¬ 
ible  components  will  be  called  two-level  systems,  etc. 

Development  testing  is  generally  regarded  as  distinct  from  and 
preceding  production  or  operational  testing.  Its  purpose  is  to  deter¬ 
mine  whether  a  particular  new  system  design  will  work  (i.e.  ,  satisfy 
given  performance  requirements,  including  endurance)  in  the  environment 
in  which  it  is  intended  to  function.  Development  testing  is  concluded 
as  soon  as  a  prototype  design  is  found  to  operate  satisfactorily  in 
this  environment.  Production  testing  in  the  usual  sense  is  then  initiated 
to  determine,  and  perhaps  to  improve,  the  reliability  of  the  production 
process  instituted  to  manufacture  the  system  in  quantity.  In  the  present 
study,  we  concentrate  exclusively  on  development  testing, 

BASIC  CHARACTERISTICS  OF  THE  DEVELOPMENT  ENVIRONMENT 

Development  in  the  area  .of  defense  takes  place  in  an  environment 
which,  although  highly  difficult  to  cope  with,  can  easily  be  described. 


Thus,  there  is  general  agreement  that  the  chief  characteristics  of 
the  environment  faced  by  the  developer  are  uncertainty,  the  necessity 
of  making  decisions  based  on  current  knowledge,  and  the  generation 
of  new  knowledge  as  the  development  proceeds.  Each  of  these  charac¬ 
teristics  is  formally  incorporated  into  the  models  constructed  in 
this  study  via  the  approach  of  modern  decision  theory. 

SETTING  OF  THE  PROBLEM 

The  problem  to  be  discussed  is  that  faced  by  the  project  director 
and  those  of  his  subordinates  and  contractors  who  must  decide  what 
tests  to  perform,  if  any,  at  each  stage  in  the  development  of  a  system. 
A  closely  related  decision  problem  concerns  the  extent  of  redesign, 
if  any,  to  be  done  once  the  results  of  a  given  test  are  known.  While 
the  project  director  and  the  test  supervisors  concerned  must,  by  ref¬ 
erence  to  a  flexible  or  conditional  listing  of  possibilities,  consider 
the  whole  future  of  the  project  in  deciding  what  to  do  next  at  a  given 
point  in  the  development,  they  cannot  specify  the  precise  sequence  of 
steps  to  be  taken  for  the  whole  project  in  advance.  This  is  because 
they  are  in  a  situation  which  is  adaptive:  the  best  action  to  take  at 
a  given  point  is  a  function  of  the  test  results  obtained  so  far. 

Accordingly,  the  models  in  this  paper  will  be  developed  with  two 
purposes  In  mind.  First,  to  provide  the  project  director  and  his 
staff  with  a  means  for  determining  the  "best”  test  to  perform  at  a 
given  stage  in  the  development.  Second,  to  enable  the  same  decision¬ 
makers  to  choose  intelligently  anong  available  actions  once  the  test 
results  are  known. 

Consider  the  abstraction  given  in  Fig.  1  of  a  typical  (but  highly 
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Key 

0  Begin  development 

1  Complete  construction  of  component  1 

2  Complete  construction  of  component  2 

3  Complete  construction  of  component  3 

4  Complete  testing  and  redesign  of  component  1 

5  Complete  testing  and  redesign  of  component  2 

6  Complete  testing  and  redesign  of  component  3 

7  Complete  assembly  of  subsystem  1 

8  Complete  testing  and  redesign  of  subsystem  1 

9  Complete  assembly  of  system 

10  Complete  testing  and  redesign  of  system 


Fig.  1  -  PERT  Network  for  a  System  Consisting  of  Three  Components 


simplified)  development  project.  Experience  indicates  that  the  major 
sources  of  uncertainty  in  development  can  be  traced  to  those  activities 
that  involve  testing  and  redesign  (i.e.,  those  that  end  in  events  4, 

5,  6,  8,  and  10).  These  activities,  then,  should  logically  be  the 
prime  targets  in  an  attempt  to  provide  a  basis  for  improved  decision¬ 
making  in  the  development  field,  and  will  be  so  viewed  in  this  paper. 

OBJECTIVE  FUNCTION 

The  objective  of  development  is  to  bring  to  "completion"  a  new 
system  or  item  not  previously  available.  "Completion"  may  be  charac- 


tericed  by  a  triplet  of  measurements:  money  cost  x,  time  elapsed  t, 
and  system  performance  r.  It  will  be  assumed  that  the  developer^  has 
preferences  over  the  various  triplets  (x,  t,  r)  that  the  system  could 
exhibit  once  it  is  "completed."  These  preferences  will  be  represented 
by  the  function  U(x,  t,  r)  which  Is  defined  for  all  non-negative  values 
of  x,  t,  and  r.  It  has  the  property  that  U(Xp  Tj)  >  U(x2,  t^,  r2> 
if  and  only  if  the  triplet  (x^,  rp  is  preferred  to  the  triplet 
(x,,  t2>  r2) .  Presumably,  IT  is  decreasing  in  x,  nonincreasing  in  t, 
and  nondecreasing  In  r. 

In  most  cases,  the  system  performance  element  of  the  triplet  would 
itself  be  a  vector  r  »  (rj,  ....  r^,  the  components  of  which  would 
correspond  to  such  magnitudes  as  speed,  range,  CEP,  weight,  reliability, 
etc.  To  simplify  the  exposition  we  assume  that  a  given  performance 
level  r  is  to  be  achieved  and  that  no  additional  utility  is  derived  by 
exceeding  r.  Thus, 

U(x,  t,  O  =  U(x,  t,  r) 

for  all  x,  t,  and  all  r'  such  that  r^  2  r,  that  is,  v'.  £  r.  for  all  i, 
and 

U(x,  t,  r*)  =  U(x,  t,  0) 

for  all  x,  t,  and  all  r"  such  that  r '  <  r. 

Let  V(x,  t)  -U(x,  t,  r)  +  K,  where  K  is  a  constant  such  that 
V(0,  0)  -  0.  Referring  to  t  as  the  time  cost,  we  may  call  V  the  time- 
and-money  cost  function. 

V  will  now  be  specified  more  precisely.  The  utility  of  money  in 

The  person  or  persons  charged  with  making  the  basic  decisions  for 
a  given  system. 
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a  large  organization,  such  as  the  government,  tends,  as  a  result  of 
the  aggregation  of  many  individuals'  functions,  to  be  linear.  Also, 
a  single  development  project  usually  requires  only  a  small  portion  of 
the  government's  total  resources,  so  that  a  nonlinear  utility  function 
would  be  well  approximated  by  its  tangent  at  some  point  in  the  relevant 
region.  Thus  we  lose  little  in  our  representation  of  the  real  world 
and  gain  much  by  way  of  simplicity  by  assuming  V  to  be  linear  in  money. 
Now  it  can  easily  be  shown  that  this  condition  Implies  that  V  is 
decomposable  and  may  be  written 

V(x,  t)  *  x  +  a(t) , 

where,  since  V  is  nondecreasing  in  t,  a  is  a  nondecreasing  function 

(not  necessarily  linear).  While  we  would  expect  a  to  be  linear  for 

most  projects  (that  is,  the  "cost"  per  time  unit  to  be  fixed),  it  is 

probably  a  sharply  increasing  function  in  the  typical  crash  program. 

In  this  Memorandum,  a(t)  will  be  assumed  to  be  identically  zero  to  keep 

the  discussion  as  simple  as  possible;  in  a  later  study  we  will  consider 

t 

both  the  money  cost  and  the  time  cost, 

PROBABILITIES 

The  required  performance  level  r  of  the  system  establishes  a  stan¬ 
dard  against  which,  at  any  point  in  time,  the  developer  may  subjectively 
derive  a  two-valued  probability  function  in  advance  for  each  part, 
component,  and  subsystem.  As  the  development  proceeds,  the  developer 

^There  is  no  reason  why  the  inverse  procedure  could  not  be  chosen. 
The  decision  to  consider  the  money  cost  rather  than  the  time  cost  first 
was  completely  arbitrary. 


will  have  numerous  opportunities  to  modify  his  initial  probabilities 
by  conducting  tests.  The  conditional  distributions  of  the  outcomes 
(messages)  of  these  tests  will  be  assumed  to  be  known.  Of  course, 
in  real  world  situations,  these  distributions  would  also  have  to  be 
determined  subjectively  since  in  development  there  is,  by  definition, 
no  directly  applicable  experience  to  fall  back  on. 

We  do  not  consider  here  the  question  of  how  to  derive  subjective 
probability  distributions.  Numerous  descriptions  of  how  this  may  be 
accomplished,  on  the  basis  of  postulates  presupposing  certain  consist¬ 
encies  in  behavior,  are  available  in  the  literature.^ 

Neither  do  we  argue  the  merits  of  using  subjective  probabilities 
in  relation  to  alternative  Inputs.  We  merely  call  attention  to  two 
relevant  observations.  As  McKean  [3]  points  out,  a  decisionmaker,  in 
taking  a  position  on  an  uncertain  issue,  implicitly  quantifies  con¬ 
siderations  which  he  refuses  to  quantify  explicitly.  Going  one  step 
further,  it  follows  as  a  theorem  that  a  decisionmaker  who  observes 
a  certain  measure  of  consistency  in  deciding  under  uncertainty  in  fact 
Imputes  a  probability  distribution  over  the  possible  outcomes,  regard¬ 
less  of  what  criterion  is  used.  This  distribution  is  such  that  if  it 

is  used  to  solve  the  decision  problem  under  risk,  it  will  give  the  same 

* 

solution  as  was  obtained  under  uncertainty  with  the  given  criterion. 

As  a  result,  if  one  is  committed  to  this  (highly  reasonable)  measure 
of  consistency,  it  would  seem  that  one  might  as  well  convert  the 
decision  problem  to  one  under  risk  by  searching  for  the  necessary 
probability  distribution(s)  , 

^See,  for  example,  the  accounts  of  Marschak  [1]  and  Timson  [2], 
*See  Ref.  4,  pp.  287-294. 
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II,  THE  FINAL  CHECKOUT  PROBLEM  WHEN  TEST  RESULTS  ARE  TWO-VALUED 


This  section  deals  with  the  problem  of  minimizing  the  m.  ney  cost 
of  a  particular  system  once  the  final  test  stage  has  been  reached.^ 

It  vil'  be  assumed  that  in  testing  a  system,  subsystem,  component,  or 
part  the  only  Information  gained  is  whether  it  works  or  doesn't  work. 

In  other  words,  the  test  information  is  two-valued. 

ONE -COMPONENT  SYSTEMS 

To  begin  with,  consider  the  case  where  a  one-level  (one -component) 
3ystem  consisting  of  M  parts  has  reached  the  final  test  stage.  The 
problem  is  how  to  conduct  the  final  test  and  any  redesign  that  may  be 
required  so  as  to  maximize  total  utility.  In  this  case,  this  is 
equivalent  to  minimizing  the  expected  cost  of  the  final  test  since 
all  other  costs  have  already  been  incurred. 

Since  the  required  performance  level  is  given,  the  outcome  of  the 
final  test  is  either  "success"  or  "failure."  The  following  assumptions 
will  now  be  made: 

Assumption  1.  The  final  teat  of  a  component  (subsystem,  system) 
will  fail  if  and  only  if  one  or  more  of  its  parts  (components,  sub¬ 
systems)  is  not  working. 

Assumption  2.  The  failure  of  one  part  of  a  component  is  indepen¬ 
dent  of  the  failure  of  any  other  part  or  combination  of  parts. 

The  test  procedure  given  in  Fig.  2  will  be  arbitrarily  selected 
for  this  model.  (Conditions  under  which  this  procedure  is  optimal  are 

^The  final  test  stage  is  considered  first  because  in  an  adaptive 
situation  the  optimal  decision  rules,  as  will  become  evident  shortly, 
are  obtained  by  working  backwards  in  time. 
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Fig.  2  -  Flow  Chart  of  Test  Procedure  for  Final  Checkout 
of  One-component  System 

given  later.)  Having  so  limited  the  problem,  the  task  becomes  to 
select  the  sequence  in  which  the  parts  should  be  tested  in  order  to 
minimize  expected  cost.  This  problem  was  first  treated  by  Johnson  [51; 
it  will  be  briefly  reviewed  here  in  the  setting  of  the  current  study. 
The  following  definitions  will  De  used: 

=  cost  of  testing  component  i 
=  number  of  parts  in  component  i 
R  =  coat  of  removing  j th  part  from  ith  component 
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E  “  cost  of  testing  jth  part  of  ith  component 
R'l  “  cost  of  returning  part  j  back  into  component  i 
p  “  3  priori  probability  that  jth  part  of  ith  component  works 
a  coat  of  redesigning  and  testing  the  jth  part  of  the  ith 
component  so  as  to  pass  jth  part  test,  given  that  it  failed 
this  test  the  first  time 

“  sequence  in  which  parts  of  ith  component  are  tested 
C^S  )  *  total  expected  cost  of  final  checkout  of  component  i  after 
initial  component  test  when  sequence  is  used 


Bij  *  \j  "  Cij  +  Rij 

qij  =  Pij 

rr. .  *  H  -  .  p., 

ij  k*j  ik 

We  now  tnr.ke  another  assumption: 

Assumption  3.  In  each  test,  p^  >  0  for  all  i  and  j. 

This  assumption  mtrely  expresses  the  common-sense  rule  that  we 
will  not  engage  in  any  test  unless  the  a  priori  probability  that  all 
parts  involved  will  work  is  positive. 

Given  an  arbitrary  sequence  S^,  we  obtain,  assuming  that  the 
system  is  not  destroyed  if  the  final  test  fails, 

Ci<Si>  =  TTil°  +  <X  -  "!!»!!  +  <kl(Cil  +  Ei> 


+  <l  ■  VBi2  +  q12(Ci2  +  V  + 


+  (1  '  niM1)BiMl  +  qiMi(CiM1  +  Ei'1  *  qiMtEiMi’ 


or 


Mi  Mi 

1  [Bij  +  +  Ei)]  ‘  I  TTijHtJ  ’  qiM  EIM. ' 

J-l  J-i 


(i)  c^sp 
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After  the  J  -  1th  part  has  been  checked  out,  the  probability  that 
the  component  test  will  fail  again  is  (1  -  n  ).  The  last  tens  of  (1) 
indicates  that  we  need  not  test  the  last  part  If  the  procedure  takes 
us  that  far  because  then  we  would  know  that  the  part  does  not  work 
and  would  Immediately  begin  to  redesign  it. 

For  any  given  ■  (p^,  . P ^  ) ,  it  is  clear  that  with  refer¬ 
ence  to  the  terms  in  brackets,  C^( S^)  is  independent  of  the  particular 
test  sequence  used.  However,  the  values  of  the  last  two  terms  are 
clearly  affected  by  the  way  the  tests  of  the  parts  are  sequenced. 

Let  us  temporarily  fix  the  final  part  in  the  test  sequence  and 
consider  sequence  s',  which  is  the  same  as  except  that  parts  j  and 
J  +  1  <  Mi  have  been  interchanged.  This  gives 


<1  - 

VB1J  +  vcij 

+  E  > 

V 

+  (1 

-  "ij+l^ijn + 

ql,j+l<‘Ci,j+l  +  EP 

-  (1 

'  nij)Bi,j+l  '  qi, 

J+l(Ci,j+l  +  Ei} 

-  (1 

*  PijPi, j+2’ 

PiM^iJ  •  qiJ(Cij  + 

Ei> 

-Bi) 

(T7ij  '  Pijpi,J+2'  ' 

‘  **  *  piMl)  "  Bi,j+1(TT 

i.J+1 

"TTi,j+2PiJBij^pi,j+l  ‘  15  "  TTi,j+lBi,j+l(1  '  pij)' 
Simplifying  further, 

(2)  Ci(si>  *  Ci/Si)  ~  *ni, j+2^'pijBijqi,J+l  +  Pi,j+l3i,j+lqij) - 
Clearly,  this  expression  is  negative  if 

*pijBijqi , 1+1  +  pi , J  +  lBi , j+lqij  >  °: 
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that  la,  if 


(3) 


Ziiiil  <  Pj 

qij 


!i,j+i 


As  a  result.  If  (3)  holds,  sequence  s'  leads  to  e  lower  expected  final 
checkout  cost  than  5^. 

Thus,  since  (3)  Is  a  transitive  relation  over  the  values  of  j ,  we 
can,  by  successively  interchanging  adjacent  parts  until  (3)  holds  for 
all  j  <  -  1,  obtain  the  optimal  sequence  for  any  given  final  part. 

Formally,  the  rule  may  be  stated  as 


(4) 


^11^11  PikBik 
j  precedes  k  if  <  J-.*-1.*. 


*11 


*ik 


the 


We  muat  now  consider  the  final  part  in  the  sequence.  Let  S,u  be 
sequence  obtained  by  applying  (4)  to  all  parts  j  *  1,  Mt<  Let 


Sik  ***  S1M  w*t*1  tlte  Part  transferred  to  the  end.  Thus  the  problem 
is  to  determine  such  that 


(5) 


im. 


min  G 


ik’ 


where 


Gik  "  ci<V  -  ci<siMl> 

^  nl,k+lPlk^Bl,k+l  +  ^  TTi,k+2Pik')Bi,k+2  + 


+  ^  TTlM1Pik)BlMi  +  Pik^  Bik  '  qikEik 

-  C(1  -  nik>Bik  +  <l  *  ni,^Bl,k+l  +  •” 


+  ^  ‘  niM4)BlM,  "  qlM1ElMi^ 
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M. 


"i,k-lBl,k-I  +  •"  *  "iM.’tH  >  +  I  " 


qlkEik  + 

M, 


'  pik5  Z  nijBlj  “  PikBik  +  "ikSik  '  qikEik  +  *W.EiH‘ 


i=kU 


i 


Simplifying, 

(6)  G 


M 


i 


1 

J=k+l 


ik  qik  L  T1ijBtj  +  Bik^ik  *  PikJ  *  qikEik  +  qiMEiHL" 


Thu*,  can  easily  be  generated,  providing  the  desired  minimizing 
index  m^. 

Let  denote  the  set  of  indices  k*  for  which  “  l.  Then  by 

(U) ,  will  be  at  the  end  of  S^,  unless  l  as  determined  by  (5), 

in  which  case  will  appear  immediately  preceding  the  final  part  in 

S,.  In  the  first  case,  1  -  n  /  “  1  -  p,,  /  *  0  for  all  k/  «  K. ,  which 
i  ’  ik  rik  i 

implies  by  (1)  that  the  testing  would  never  reach  the  parts  in  K^; 

letting  -  K^,  we  obtain  C^(Sj)  *  Thus,  in  this  case, 

it  is  immaterial  whether  we  oxclude  from  the  test  sequence. 

In  the  second  case,  1  -  tt  ,  *  l  -  n  “  1  -  p.  so  that 
’  ik  im^  *10^ 

ci'5t>  ■  W  •  <l  •  "t.  >  I  hk- 

1  k'«. 


or 


(7) 


cl(3i)  <  Cl(Si). 
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Since  we  clearly  do  not  want  to  test  parts  that  we  know  are  work¬ 
ing,  we  must  therefore  explicitly  exclude  them  in  defining  the  test 
sequence,  Thl*  gives 

Theorem  1.  To  ensure  that  parts  known  to  be  working  are  not  tested 
for  any  component  i,  It  is  necessary  to  limit  the  domain  of  to  those 
Indices  j  for  which  p  <1,  j  =  1,  , M^. 

The  main  result  may  new  be  stated  as 


Theorem  2.  Foi 


?onent  i,  number  the  parts  for  which  <  1 


in  ascending  order  of  p^B^/q^,  j  *  then  transfer  the  mth 

element  of  to  the  final  position  M '  where  m^  is  given  by  ” 


fflink  Gik*  k  “  ••• .  M'. 


and  G, 


Gik  “  qik  T  TTijBij  +  Bik(TTik  ‘  Pik^  ‘  qikEik  +  qiM/EiM/* 


J-k+1 


i  ‘"l 


Then  this  is  the  optimal  sequence. 

★ 

Let  us  call  the  optimizing  sequence  S^.  Then 


C  (S  )  -  min  C  (S.), 

V'5i 

where  ^  is  the  set  of  all  possible  sequences  for  component  i.  Clearly 
*  — 

is  a  function  of  p^,  the  probability  vector  of  the  component,  so  that 

★  —  -if  — 

we  should  write  S^Pj) .  It  will  be  convenient  to  define  C^Cp^)  * 

•Jf  _  •• 

C^(S^(p^) ) ;  thus  the  optimal  expected  checkout  cost  when  the 

given  test  procedure  is  used  and  p^  holds. 


MULTI -COMPONENT  SYSTEMS 

We  now  consider  two-level  systems  and  find  that  the  problem  remains 
exactly  the  same,  as  shown  by  Johnson  [51.  The  test  procedure  is  depicted 
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in  Fig.  3.  We  uae  Che  convention  of  testing  a  given  component  even 
after  the  last  part  has  been  redesigned,  although  this  is  not  strictly 
necessary. 

Let  i  denote  the  ith  component  of  the  system,  using  an  arbitrary 
numbering  scheme,  where  i  "  1,  K. 

We  now  need  the  following  definitions: 

E  “  cost  of  performing  system  test 

*  cost  of  removing  ith  component  from  system 

R?  "  cost  of  returning  working  component  i  into  system 
Mi 

*  II  ^  “  *  priori  probability  that  ith  component  works 

S  “  sequence  in  which  components  are  tested 

C(S)  *  total  expected  cost  of  final  checkout,  excluding  initial  test 
of  system,  when  the  components  are  tested  in  the  sequence  S 

Bi  “  Ri  +  Ei  +  Ri 
qt  -  1  -  P4 
ttK 

"i  "  Vi  pi 


We  now  obtain 


C(S)  “  On  + 


+  •••  +  (i  . 


/C1(P  !>  \  (W  \ 

(1  -  "i>Bi +  +  e;  +  (1  -  vb2  •  +  7 

*  <[  -  Vbk  *  *  )  -  ’A- 


C(S)  -  l  (B.  +  c*(pt)  *  ,,£)  -  l  Vl  -  ,kek. 


Since  this  is  analogous  to  (1) ,  we  obtain,  by  the  proof  of  Theorems  1 
and  2,  Theorems  3  and  4. 


Fig.  3  -  Flow  Chart  of  Test  Procedure  for  Final  Checkout 
of  Multi -component  System 
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Theorem  3,  To  ensure  that  eofaponents  known  to  bo  working  are  not 
tested.  it  is  necessary  to  limit  the  domain  of  S  to  '  ose  Indices  1 
tor  which  <  1,  i  ~  1,  ...»  K. 

Theorem  4.  Number  the  components  for  which  <  l  in  ageending 
order  of  *  *  1*  *  then  transfer  the  mth  element  of  S  to 

the  final  position  K '  where  a  is  given  by 


G  *  min  G,  ,  k  *  1,  ....  K* 
m  ,  k’  ’ 

k 

and 

K  ' 

Gk  =  qk  Z  +  Bk^nk  “  Pk^  "  PkEk  +  pk/Ek' 

i-k+1 

•k 

in  analogy  with  (8).  Then  this  is  the  optimal  sequence  S  . 

—  —  —  Hr  —  ★  — 

Let  p  =  (p^,  ....  pK)  and  let  C  (p)  =  C(S  (p' ) .  Then  the  minimum 
expected  final  checkout  cost,  including  the  initial  system  test,  becomes 


(10) 


f , (p)  «  E  +  C*(p). 


^  Hr  Hr 

Clearly  for  p  -'  I,  where  I  is  the  unit  vector,  C  (I)  “  0,  Also,  C  (p) 
appears  to  be  continuous  for  0  <  p  <  I  (where  0  is  the  zeto  vector)  with 
discontinuities  at  =  1  for  all  i  and  j,  and  strictly  decreasing  in 

pij  f°r  311  if  ^ 

It  is  readily  seen  that  the  preceding  results  generalize  so  as  to 
apply  to  systems  of  any  number  of  levels. 


LOSS  OF  SYSTEM 

So  far,  we  have  assumed  that  failure  in  the  final  te*.t  of  a.  item 
does  not  result  in  loss  of  the  item.  We  now  consider  the  situation 
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where  this  is  the  case;  let  the  cost  of  a  duplicate  system  be  L  and  the 
cost  of  a  duplicate  of  component  i  be  L^,  Then 

C(S)  *  TijO  +  (1  -  tt1)(L1  +  E,  +  R')  +  C* (pp  +  qx(E  +  L) 

+  (1  -  VCL2  +  E2  +  RP  +  C2(*2}  +  q2(E  +  +  *“ 

+  (1  -  V*<V  +  Ek,  +  R^/)  +  C*r(jK,) 

+  qR/(E  +  L)  -  Pk/Ek/, 
or 

K/ 

(10a)  C(S)  =  £  (Lt  +  Et  +  r'  +  C*(pt)  +  qi(E  +  L» 

i»l 

k' 

•  I  Vh +  -i +  Rt'>  -  vV- 

i=l 

This  is  analogous  to  (9)  with  R^  replaced  by  L^,  and  E  replaced  by  E  +  L. 
Thus  Theoremt,  2  and  4  also  hold  when  the  object  tested  is  destroyed  in  the 

t 

final  checkout  as  long  as  is  replaced  by  ;  that  is,  ~  +  E^,  +  R^. 

THE  TEST  PROCEDURE:  SUFFICIENCY  CONDITIONS 
FOR  OPTIMALITY 

So  far,  we  have  taken  the  test  procedures  shown  in  Figs,  2  and  3 
as  given.  Although  chosen  arbitrarily,  as  pointed  out  earlier,  the 
procedure  appears  rather  reasonable.  We  now  give  the  conditions  under 
which  it  is  optimal.  While  these  conditions  will  be  stated  for  one-level 
systems  only,  they  essentially  "generalize"  when  multilevel  systems  are 


invo lved. 
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Wc  first  prove  the  following: 
Lemma.  Let 


(11) 


Pj*l 
1  -  P, 


pi+iBi+i  , 
1  -  pi+i 


i  ‘  i. 


K'  -  1, 


where  £,  B  >  0,  0  <  <  1,  i  “  1,  K*.  Ttan 

K' 

(12)  ff  ?i  +  e)  -  E  2  0,  J  -  1,  ....  K' 

i-1 

i/j 

implies  that 

K' 

(13)  Pk(Bk  ff  Pt  +  E)  -  E  2  0,  j  “  1,  ....  k', 

i*l 

i^j  k  -  2,  ....  k'. 


Proof.  For  any  k  *  2,  ....  K/,  (11)  gives 


(14) 


plBl  4 


Pk\(1  '  Pl> 

1  -  Pt.  ' 


The  inequality  (12) 
is  substituted  for 

pkBk(I  -  P1> 
1  -  pk 


Is  then  preserved  if  the  right-hand  side  of  (14) 
PjBj  in  (12),  giving 

k' 

'  ff  pi  +  plE  '  E  2  °»  J  "  1 . K'« 

i-1 


l" 


which  reduces  Co  (13)  upon  factoring  out  1  -  p 
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Theoretn  5.  For  the  teat  procedure  In  Pig.  2  to  be  optimal.  It  Is 


sufficient  that 


n('i  ff  "t +  E)  - ' 4 


i  m  i.  ,  k\ 


where  the  subscript  1  denotes  the  ith  part  and  the  parts  have  been  num¬ 
bered  in  ascending  order  of  p^B^/O  -  p^). 

Proof.  We  first  show  that  the  procedure  in  Fig.  2  is  optimal 
Riven  that  a  system  test  is  performed  first,  and  then  show  that  a  sys¬ 
tem  test  should  indeed  be  performed  as  the  first  step. 

(1)  Given  that  the  initial  system  test  fails,  an  alternative  test 
procedure  must  necessarily  encompass  the  testing  and  repairing  of  two 
or  more  parts  before  a  new  system  test  is  made.  Let  Q(P)  be  the  total 
checkout  cost  when  parts  a,  . ...  a/  are  grouped  together  somewhere  in 
the  test  sequence  followed  by  parts  s'  +  1,  K#  (separably  or  in 

groups).  Similarly,  let  Q(p')  be  the  total  checkout  cost  when 
k  e  fa,  ....  s']  is  "broken  out"  of  the  group;  that  is,  after  the 
other  parts  in  the  group  (but  before  parts  s'  +  1,  ...,  K#)  have  been 
tested,  a  system  test  is  performed  to  determine  whether  further  test¬ 
ing  is  needed.  We  then  obtain 


(15)  Q(P)  -  Q(P')  -  •“  +  (1  -  p  .  ....  p  /)  (B  +  '••  +  B  ..) 

®  K  I  ft 


♦  <l  -  >.>c.  +  •••  +  <*  -  V>v 


+  O  -  P. . p#')£ 
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•  +  C1  "  P3 . PK#)(B8  +  +  Vl 

+  Vi  +  ***  +  V*  +  -  p8>c,  + 

+  (1  -  Pa*'ctr  +  (1  *  Pg . pk-lpk+l*  **'*  P»/5E 

+  (1  •  PkV+l>  **’  V)Bk  +  (l  ‘  Pk)E] 

*  (1  '  V  ’  pk-lpk+i*  ****  ?a^ 
k' 

■  W\  rf  ^ +  *)  -  «]■ 

i-a  '+1 

Since  the  first  factor  of  (15)  is  always  positive  and  the  second  factor 
is  non-negative  whenever  (12)  is  by  the  lemna,  it  follows  that  part  k 
should  be  "broken  ou‘"  whenever  (12)  holds,  and,  since  k  may  be  any  part, 
that  every  part  in  every  group  should  be  "broken  out."  Consequently,  the 
test  procedure  in  Fig.  2  is  optimal  given  that  a  system  test  Is  performed 
as  the  first  step. 

(2)  L-et  C(?i)  be  the  total  final  checkout  cost  when  the  test  pro¬ 
cedure  in  Flg„  2  is  used,  except  that  no  system  test  is  performed  until 
after  the  ith  part  test.  We  then  obtain 

(16)  C(P1)  -  C(PQ)  -  Bx  +  (1  -  PjK,  +£  +  ••• 

-  [E  +  (1  -  px . PK^®1  +  (l  "  ppCl 

+  (1  -  PX)E  +  •*•] 

*  P^(Pj*  •••*  ^  E)  -  E, 


which  is  non-negative  whenever  (12)  is.  Similarly, 


1  . 


i 


(17)  C(Pt)  -  C(P1_1)  -  pt(pi+1,  ....  +  E)  -  E, 

1  “  2,  ....  k', 

which,  by  the  lemma,  1*  non-negative  whenever  (12)  is.  Consequently, 
the  theorem  holds. 
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III,  MULTISTAGE  TESTING 

i  — 

Since  the  minimum  expected  cost  of  the  final  checkout,  C  (p) ,  is 
a  function  of  the  a  priori  probabilities  that  the  various  parts  are 
working,  we  aie  now  in  a  position  to  assess  the  value  of  performing 
tests  of  parts,  components,  and  subsystems  at  earlier  stages  in  the 
development.  Before  constructing  an  analytical  model  to  deal  with 
this  problem,  several  important  aspects  of  its  setting  must  be  clarified. 

We  have  already  pointed  out  that  each  element  of  p  expresses  the 
a  priori  probability  that  a  particular  part  will  perform  its  job  in  the 
final  system.  Thus  at  any  one  point  in  time,  the  same  part  may  have 
an  entirely  different  a  priori  probability  of  working  in  one  system  as 
opposed  to  another. 

In  effect  Assumption  1  states  (perhaps  somewhat  idealistically) 
that  the  tests  used  for  the  final  checkout  provide  perfect  information. 
Letting  z  denote  the  state  of  the  system  and  y  the  message  of  the  test 
as  to  what  that  state  is,  perfect  information  implies  the  following 
conditional  distribution  P(y/z) : 


P(y/z) 

Message 

1 

State  of  System 

Working 

yl 

R 

P(y/z) 

Working  (zp 

1 

0 

l 

Not  working  (z^) 

0 

l 

l 

When  a  test  is  performed  at  an  earlier  stage  in  the  development, 
we  would  not  expect  to  obtain  a  conclusive  answer  as  to  whether  the 
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part,  component,  or  subsystem  tested  will  work  In  the  final  system. 
Thus  we  might  expect  to  have  a  situation  like  this; 


P<y/*)  1 

Message  j 

State  of  Part 

*1 

y2 

£v  P(y/0 

Z1 

1 

0 

l 

z2 

1/4 

3/4 

i 

In  this  case,  P(yj/*^)  “  1  and  P(y^/z^)  “  0.  This  implies  that  if  the 
part  is  working  the  test  will  bear  this  out,  and  in  most  real  situations 
this  will  probably  be  true.  However,  if  we  wish,  for  example,  to  take 
into  account  the  possibility  that  the  test  equipment  itself  may  be 
faulty  without  our  discovering  it,  we  obviously  have  a  situation  in 
which  P(y2/zj)  >  0. 

In  the  example,  P(y2/Z2^  “  3/4;  that  is,  if  the  part  is  not  working, 
the  probability  that  the  test  will  discover  this  fact  is  3/4. 
may  be  considered  as  a  measure  of  the  "power"  of  the  test.  In  general, 
the  earlier  the  stage  in  which  the  test  is  made,  the  smaller  we  would 
expect  its  "power"  to  be. 

For  a  test  to  have  any  value,  its  results  must  provide  learning, 
that  is,  for  a  modification  of  the  a  priori  probabilities  Involved. 

The  mechanism  by  which  this  modification  is  accomplished  is  provided 
by  Bayes'  Theorem,  Letting  P(z)  denote  the  a  priori  probability  of 
state  z,  we  obtain  by  this  theorem 


P(*/y) 


p(y/rip<Q 
P(y /*)?(*)• 
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In  the  above  example,  letting  P(z^)  a  1/2, 


Ksj/yj) 


P<yl/z1)P(z1) 

P(y17z1)P(z1)  +'¥(y1/z2)Hr2) 


i  :  1/2 

1  *  1/2  +  1/4  *  1/2 


4/3 


pf,  \  u  _ L/4  ‘  1/2 _  m 

U2/yl'  1  *  1/2  +  1/4  *  1/2  i/5‘ 


,  p(y2/zi)p<zt) 

P(z  /v  ^  —  - - - -  -  f  *  _  - 

1  yr  P(y2/2l)P(Zl)  +  P(y2/Z2)p(z2)  °* 


P(z2/y2)  ■  1  -  Pfzj/y^  -  1. 


Thus,  on  the  basis  of  this  test,  we  would  revise  the  probability 
that  the  part  is  working  from  1/2  to  4/5  if  message  y^  is  received, 
and  to  0  if  y2  is  obtained.  Note  that  if  P(y/z)  *  x^,  say,  for  all  z, 
the  a  posteriori  probabilities  would  coincide  with  the  a  priori  prob¬ 
abilities.  Therefore,  for  learning  to  take  place,  the  test  must  generate 
messages  that  are  in  some  way  correlated  with  the  relevant  states  of 
the  part  tested. 

When  y2  Is  received,  it  will  frequently  seem  logical  to  take  some 
corrective  action  in  the  form  of  rework  or  redesign,  since  this  would 
normally  be  cheaper  than  to  do  so  when  the  part  has  already  become  part 
of  a  larger  unit.  However,  the  same  action  appears  rather  illogical 
when  message  is  received  in  the  test.  On  the  other  hand,  the  imper¬ 
fect  nature  of  the  information  gained  in  early  testing  tends  to  offset 
the  advantages  of  early  redesign.  Thus,  we  have  a  "dual"  problem: 
what  teat  to  perform  on  the  given  part  and  what  corrective  action  to 
take  once  the  result  is  known. 
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P RE -CHECKOUT  TESTING  OF  PARTS 

Let  us  stake  the  following  definitions: 

f>  *  a  priori  probability  vector  of  system 
p'  m  a  posteriori  probability  vector  of  system 
y  "  a  possible  test  result 

Y  “  the  set  of  possible  test  results  of  an  available  test 
Q}  *  the  set  of  available  tests  (includes  null  test,  Y^) 
a  *  a  corrective  action 

A  =  the  set  of  corrective  actions  (includes  null  action,  a q) 

T(Y)  “  cost  of  performing  test  Y 

Cy(y,  a)  “  expected  cost  of  taking  corrective  action  a  when  message  y 
is  received  and  teat  Y  is  used 

y/(y,  *»)  *  transformed  message  y*  after  action  a  has  been  taken  follow¬ 
ing  receipt  of  message  y 

Let  us  now  discuss  some  properties  of  Cy  and  yf.  Clearly, 

Cy(y,  Sq)  “  0  for  all  y,  and  y'(y,  a^)  *  y  for  all  y  since  "no  action" 
costs  nothing  and  is  incapable  of  performing  a  transformation  of  a 
message.  Returning  to  our  example  in  which  is  the  message  that  the 
part  works  and  y^  that  it  does  not,  let  a ^  be  the  action  consisting  of 
redesigning  the  part  until  It  passes  the  test  (message  is  obtained). 

We  then  obtain  y^y^,  aj)  =  y^.  In  general,  the  effect  of  taking  a  cor¬ 
rective  action  other  than  a^  is  to  transform  a  "bad"  message  into  a 
"good"  one  while  at  the  same  time  incurring  a  cost. 

We  can  now  state  the  problem  in  which  we  have  at  some  point  an  oppor¬ 
tunity  to  test  and  redesign  one  part  before  the  final  checkout  stage  is 


reached: 


(18)  min  {t(Y)  +  7  P(y)  min  [c  (y,  a)  +  C*(p'(p,  y'(y,  a)))]}. 
Ye<ty  “  acA 
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As  stated  earlier,  the  problem  is  two-fold.  First,  for  a  given 
test  and  test  result  we  Bust  find  the  least-cost,  or  "best,1*  action  «. 

Then,  weighting  this  cost  by  the  probability  of  receiving  the  given 
message  and  suamsing  over  all  messages,  it  remains,  after  repeating 
this  procedure  for  all  tests  Y,  to  pick  the  test  with  the  least  ex¬ 
pected  cost.  The  "best"  test  may,  as  stated  earlier,  be  the  null  test. 

★ 

The  function  ,  among  all  possible  «  which  associates  with 
Y»P  *>P 

each  possible  y  €  Y  the  action  a  c  A  that  minimizes  the  bracketed  ex¬ 
pression  of  (18)  is  the  decisionmaker's  best  decision  rule  for  that 

—  * 

test  and  a  priori  probability  vector  p.  Letting  Y  be  the  best  test 
for  the  part  that  can  be  tested  prior  to  the  final  checkout,  the  minimum 
expected  test  and  redesign  cost  of  the  system  Is 

(19)  T(Y*)  +  Y  P(y)[c  *(y,  o**  _(y))  +  C*(p'(p.  y'(y,  __(y»))]. 

y eY*  V  Y  ,P  Y  ,P 

Since  the  minimum  expected  test  and  rework  cost  of  the  system  when 

A  — >» 

the  part  is  not  tested  prior  to  the  final  test  Is  C  (p) ,  the  value  of  the 
information  gained  by  conducting  test  Y  is  clearly 

(20)  C*(p)  -  T(Y)  -  £  P(y)[cY(y,  **  -<y))  +  C*(p'(p,  y'(y,  a*  -(y))))l 

yeY 

that  is,  the  amount  by  which  the  minimum  total  expected  cost  of  testing  is 
reduced  when  test  Y  is  employed. 

PRE -CHECKOUT  TESTINC  OF  COMPONENTS 

Just  as  opportunities  exist  to  test  parts  before  they  are  assembled 
into  components,  opportunities  are  usually  available  to  test  components 
before  they  are  assembled  into  subsystems,  to  test  subsystems  before 
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assembly,  etc.  Here  we  need  to  draw  on  the  preceding  discussion  of 
pre-checkout  testing  of  parts  as  well  as  the  discussion  In  Sec,  IX. 

Let  be  the  probability  that  the  ith  component  works  before  the 
teat  (of  the  component)  and  be  the  a  posteriori  probability.  By 
Assumption  2, 

Mi 

pi  *  ft  Pu¬ 
ri 


Once  the  result  of  the  test  is  known,  can  now  be  obtained,  as  before, 
by  Bayes'  Theorem.  Again  by  Assumption  2, 

M, 

_  / 


n  p,v 


j-i 


In  taking  corrective  action  in  response  to  a  message  y  which  relates 

to  a  component,  we  again  assume  that  the  procedure  in  Fig.  3  is  used. 

Thus  the  cost  of  taking  corrective  action  becomes  a  function  of  as 

well  as  of  the  message  y  and  the  action  a.  The  minimum  expected  such 
★  — 

cost,  Cy(y,  a,  P^) ,  would  be  derived  by  the  method  of  Sec.  II. 

The  value  of  the  information  gained  by  conducting  pre-checkout  test  Y 
of  component  i  (when  other  components  and  parts  are  not  so  tested)  is 
then,  in  analogy  with  (20), 


(21)  C  (p)  -  T(Y)  -  y  P(y)[c  (y,  a*  -(y) ,  p)  +  C  (p'(p,  y'(y,  o*  ~(y))))]. 

1  *  )r  1  *  P 

yeY 

The  minimum  expected  test  and  rework  cost  of  the  system  now  becomes 

(22)  min  {t(Y)  +  >  P(y)  min  [cv(y,  a ,  p)  +  C  (p'(p,  y'(v. 

Ys<y  yeY  aeA  ‘ 
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TKE  COKt’LKTE  PROBLEM 

Having  formulated  the  test  and  lework  problem  for  two  levels,  we  an 
now  in  a  position  to  state  the  entire  development  problem  in  sequential 
form.  Consider  Fig,  4  in  which  the  test  stages  involved  in  the  development 
of  a  two- level  system  are  indicated.  Any  activity,  or  the  smallest  set 
of  activities,  which  can  be  finished  before  all  remaining  activities  must 
begin,  constitutes  a  stage.  Thus,  a  stage  is  the  smallest  set  of  activ¬ 
ities  during  which  information  that  can  be  used  for  subsequent  stages  can 
be  obtained. 

It  is  now  clear  that  at  any  stage  the  choice  of  a  test  is  often  a 
choice  of  a  combination  oi  tests;  we  therefore  use  the  word  "test”  to  refer 
to  a  particular  combination  of  tests  (see,  for  example,  stage  I  in  Fig.  4). 

We  now  make  the  following  definitions: 

N  =  number  of  stages  at  which  tests  can  be  performed 
£^(p)  =  minimum  expected  cost  of  developing  system  when  tests  exist 
at  N  stages 

Y^  =  the  set  of  possible  test  results  of  a  te3t  available  at 
Nth  stage  from  end 

‘iJ  =  the  set  of  tests  available  at  Nth  stage  from  end  (includes 
null  test) 

A  =  the  set  of  redesign  actions  available  at  Nth  stage  from  end 

N 

T(Y^)  *  cost  of  performing  test  Y^ 

*  — 

C  (y,  a,  p)  *  minimum  expected  coat  of  taking  corrective  action  a  when 

yn 

test  is  used,  test  result  y  Is  received,  and  the  a  priori 
probability  vector  is  p 

K  -  cost  of  pioducing  and  assembling  parts  and  com;  merits  ex¬ 
clusive  of  testLng  and  redesign  activities 


-2<> 
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Fig.  4  -  Test  Stages  of  a  System  Consisting  of  Three  Components 


Extending  our  two-stage  model  to  N  stages,  we  obtain  for  the  minimum 
expected  cost  of  developing  the  system,  by  the  principle  of  optimality  [6,  p,  83], 


(23)  fN(p)  =  K  +  min  {t(Yn)  +  ^  P(v)  min  [cy  (y,  a,  p) 

YN€%  yeYN 


aeA.T  ~N 
N 


*N_i<.p'(p,  y '(y,  a))) ]}. 


where  f1(p)  =  C*(p) . 

At  any  stage,  this  equation  enables  the  developer  to  choose  the  best 
test  for  that  stage.  (Remember  that  the  best  test  may,  of  course,  be  no 
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teat.)  Since  the  best  decision  at  any  stage  always  depends  on  past 
actions  and  test  results,  the  model  is  an  adaptive  one. 

AS  EXAMPLE 

At  this  point,  we  provide  an  example  to  illustrate  the  use  of  the 
preceding  model.  Let  the  task  be  to  develop,  at  minimum  cost,  a  new 
missile  system  (system  X),  which  is  to  meet  a  given  performance  level. 
For  purposes  of  illustration,  we  assume  that  system  X  consists  of  only 
two  subsystems  (parts) --a  guidance  subsystem  and  a  propulsion  subsystem. 
The  PERT  chart  governing  its  development  is  given  in  Fig.  5,  where  the 
two  possible  test  stages  are  indicated. 


K SI 


0  Begin  development 

1  Complete  construction  of  guidance  subsystem 

2  Complete  construction  of  propulsion  subsystem 

3  Complete  testing  and  redesign  of  guidance  subsystem 

4  Complete  testing  and  redesign  of  propulsion  subsystem 

5  Complete  assembly  of  total  system 

6  Complete  testing  and  redesign  of  total  system 


Fig.  5  -  System  X 

The  relevant  Information  concerning  the  final  (second)  test  stage 
is  sussaarized  in  Table  1.  Tables  2  through  9  and  the  text  following 
contain  the  necessary  information  required  for  decisionmaking  with  re¬ 
spect  to  test  stage  1. 


Table  1 


COSTS  RELATING  TO  FINAL  TEST  STAGE 


Mtaaile  System  X: 

E  »  1  Expected  coat  of  performing  final  teat  firing  of  one 

missile. 

L  ■  7  Expected  cost  of  replacing  a  system  used  up  in  a 

final  test  firing. 

Guidance  Subsystem  {Subsystem  1): 

E ^  “  10  Expected  cost  of  tests  required  to  be  performed  on 

guidance  subsystem  to  ascertain  whether  it  meets  the 
required  performance  level  (includes  the  production 
costs  of  guidance  and  auxiliary  subsystems  used  up  in 
these  tests),  given  that  the  final  system  test  failed. 

L  *  4  Expected  coat  of  (replacing)  guidance  subsystem  (used 

up  in  a  final  system  test  firing) . 

R.'  “  1  Expected  cost  of  integrating  guidance  subsystem  with 

propulsion  subsystem  to  fora  total  system. 

Cj  “  8  Expected  cost  of  redesigning  and  testing  guidance  sub¬ 

system  to  pass  the  guidance  subsystem  test,  given  that 
this  test  was  not  passed  the  first  time  (Includes  the 
cost  of  replacing  guidance  and  auxiliary  subsystems 
used  up  in  this  phase) . 


Propulsion  Subsystem  (Subsystem  2): 


“  12  Expected  cost  of  tests  required  to  be  performed  on  pro¬ 

pulsion  subsystem  to  ascertain  whether  it  meets  the 
required  performance  level  (Includes  the  production 
costs  of  propulsion  and  auxiliary  subsystems  used  up 
in  these  tests) ,  given  that  the  final  system  test 
failed. 


L 


2 


2 


r'  -  r; 


Expected  cost  of  (replacing)  propulsion  subsystem  (used 
up  in  a  final  system  test  firing). 


Expected  cost  of  redesigning  and  testing  propulsion 
subsystem  to  pass  the  propulsion  subsystem  test,  given 
that  this  test  was  not  passed  the  first  time  (includes 
the  cost  of  replacing  propulsion  and  auxiliary  sub¬ 
systems  used  up  in  this  phase) , 
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Table  2 

TESTS  AVAILABLE  AT  TEST  STAGE  1 


Configuration 

a 

Designation 

No 

Test 

Centrifuge  Tost 
of  Guidance 
Subsystem 

Sled  Test 
of  Guidance 
Subsystem 

Static  Firing  Test 
of  Propulsion 
Subsystem 

Coat 

T(Y) 

Yo 

X 

0 

Y1 

X 

.1 

Y2 

X 

2 

Y3 

X 

1 

Yl,2 

X 

X 

2.1 

Yl,3 

X 

X 

l.l 

Y2,3 

X 

X 

3 

Yl,2,3 

X 

X 

X 

3.1 

g 

For  simplicity,  in  the  case  of  Y^  we  assume  that  Y^  and  Y^  provide 
independent  messages.  * 


Table  3 

PROPERTIES  OF  TEST  Yj 
(Centrifuge  Test  of  Guidance  Subsystem) 


Probability  of  Receiving  Message  y 

Given  State  z  (P(y/z)) 

State  of  Subsystem 
with  respect  to 

System  X  Requirements 

Test  Passed 
at  High  Speed 

Test  Passed  at 
Medium  Speed  Only 

(y2) 

Test 

Failed 

<y3> 

Working  (z  ) 

.8 

.2 

0 

Not  working  (Zj) 

.2 

.u 

.4 
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Table  4 

PROPERTIES  OF  TEST  Yj 
(Sled  Test  of  Guidance  Subsystem) 


State  of  Subsystem 
with  respect  to 
System  X  Requirements 

Probability  of  Receiving  Message  y 
Given  State  z  (P(y/z)) 

Test  Passed 

(yp 

Test  Failed 

(y2} 

Working  (zp 

.9 

.1 

Not  working  (zp 

.3 

.7 

Table  5 

PROPERTIES  OF  TEST  Y3 

(Static  Firing  Test  of  Propulsion  Subsystem) 


State  of  Subsystem 
with  respect  to 
System  X  Requirements 

Probability  of  Re 
Given  State 

ceiving  Message  y 

z  (P(y/z)) 

Test  Passed 

Test  Failed 

(y2> 

Working  (zp 

.95 

.05 

Not  working  (z*) 

2 

.45 

.55 

-34- 


Table  6 

PROPERTIES  OF  TEST  Y 

1,2 

(Centrifuge  Test  end  Sled  Test  of  Guidance  Subsystem) 


Probability  of  Receiving  Message  y 
Given  State  z  (P (y/z)) 


State  of  Subsystem 
with  respect  to 
System  X 
Requirements 

Centr:  y^ 
Sled:  y^ 

^l.P 

Centr:  y^ 
Sled:  y2 

<yl,2? 

Centr:  y^ 
Sled:  y^ 

(y2.1> 

Centr:  y2 
Sled:  y2 

(y2,2) 

Centr:  y3 
Sled:  y^ 

fy3,P 

Centr:  y3 
Sled:  y2 

(y3>2) 

Working  (zp 

.72 

.08 

.18 

.02 

0 

0 

Not  working  (*2) 

.06 

.14 

.12 

.28 

.12 

.28 

Table  7 

COST  OF  AVAILABLE  REDESIGN  ACTIONS  AS  A  FUNCTION  OF  MESSAGES  RECEIVED: 

TESTS  Yj,  Y2,  AND  Y3 


Cost  of  Redesign  Action  (c^(y,  a)) 


Action  Taken  on  Conclusion  of  Test 

Test 

Test 

Y2 

Test 

Y3 

Message 

Message 

Message 

yi 

H 

H 

B 

B 

B 

y2 

aQ  None 

0 

0 

0 

0 

0 

0 

0 

a^  Redesign  to  obtain  y^ 

-- 

B 

8 

■ 

5 

B 

6 

a2  Redesign  to  obtain  y2 

H 

3 

■ 

-- 

B 

-- 
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Tab  lc  8 

COST  OF  AVAILABLE  REDESIGN  ACTIONS  AS  A  FUNCTION  OF  MESSAGES  RECEIVED: 

TEST  Y. 

'  *  » 


Cost  of  Redesign  Action  (Cy(y,  a)) 

jj- 

K 

Message  Received 

AVAILABLE  REDESIGN  ACTIONS:  TES 
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Thc  costs  of  the  redesign  actions  given  in  Table  9  are  obtained 
from  Tables  7  and  8  by  simple  addition  since  and  Y^ „  Y,  end  Y^, 
and  Y^  j  and  Y^  apply  to  different  subsystems.  Thus,  for  example. 


( y ■> 


6+6-12. 


f  VJ3  2  2*  2  1  I'  Y  2*  2  ]/  wy  'J2 

1,2,3’  ’  1»2  ’  3 


)  +  cv  (y0 ,  Sj) 


From  B,  “  E.  +  L.  +  R/,  we  obtain  that  8.  *  15  and  that  B.  *  15. 

i  i  i  i’  1  2 

Applying  Theorem  2  to  (1),  we  find  by  reference  to  (10)  that  the  minimum 
expected  final  checkout  cost  of  system  X  at  the  beginning  of  the  final 
test  stage,  P2) »  is 

(24)  fl^Pl-  p2^  *  61  '  16pl  ‘  l4p2  "  15plp2 

+  min  {*3p2  -  12,  -Spj^  -  10}. 

For  example,  when  (the  probability  as  of  the  beginning  of  the  final 
test  stage  that  the  guidance  subsystem  will  work)  is  ,5,  and  (the 
corresponding  probability  for  the  propulsion  subsystem)  is  .5,  we 
obtain 

f ^( . 5 ,  .5)  =  28.75,  S*  =  (1,  2), 

and  when  ■  .9  and  ■  .7, 

fjC.9,  .7)  -  12.85,  S*  •  (2,  1). 

Thus,  in  case  the  first  test  firing  of  the  missile  fails,  the  optimal 
sequence  in  the  first  case  is  to  perfect  the  guidance  subsystem  before 
attempting  another  firing  of  the  full  system  and,  if  necessary,  proceed 
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t o  redesign  Che  propulsion  subsystem.  In  the  second  case,  the  reverse 
sequence  should  oe  used. 

The  total  minimum  expected  cost  of  testing  and  redesign  for  each 
of  the  possible  tests  that  say  be  used  In  stage  1,  as  of  the  beginning 
of  that  stage  (assuming  an  optimal  test  sequence  In  stage  2),  Is  shown 
In  Table  10  for  selected  probability  vectors  p.  When  p  “  (.5,  .5),  we 
note  that  test  Yj  ^  j  gives  the  lowest  expected  cost:  f2(.5,  .5)  “  20.63. 
Analogously,  we  note  that  fj(.5,  .95)  *  11.06  (Yj  2  optimal),  f2(,9,  .7)  * 
10.42  (Yj  3  optimal),  and  f2(.95,  .95)  -  3.61  (Yt  optimal).  It  should 
be  noted  from  Table  10  that  the  null  test  (Y2)  is  the  worst  test  when 
p  “  (.5,  .5),  while  it  is  the  second  best  test  when  p  *  (.95,  ,95). 

The  best  corrective  actions  crY  — (y)  for  the  best  tests  in  Table  10 
are  given  in  Table  11.  Results  for  tests  Yj  and  Y ^  3  illustrate  that  it 
Is  not  always  optimal  to  redesign  so  as  to  pass  a  given  test  on  all  counts 
when  the  outcome  of  the  first  test  trlsl  is  unfavorable. 

Let  us  illustrate  briefly  how  f2(,9,  .7)  *  10,42  was  computed  for 
test  Yl>3. 

Step  1.  Bayes'  Theorem  gives,  by  direct  application  of  the  condi¬ 
tional  probabilities  in  Tables  3  and  5, 


-/  -  /  -8Pl  -95p2  \ 

p  (yl,l*  P)  *  \.BPl  +  .2(1  -  Pl)*  .95p2  +  .45(1  -  p2)y* 

so  that  p'(yj  ^ ,  (.9,  .7))  “  (.97,  .83).  Inserting  this  value  in  (24) 
we  obtain  fj(.97,  .83)  3  6.93.  In  the  same  way,  we  obtain  for  the 


39 


Table  10 

TOTAL  MINIMUM  EXPECTED  COST  OF  TESTING  AND  REDESIGN  OF  SYSTEM  X, 
AS  OF  THE  BEGINNING  CF  THE  FIRST  TEST  STAGE, 

AS  A  FUNCTION  OF  THE  TEST  USED  AT  THE  FIRST  STAGE 
FOR  SELECTED  PROBABILITY  VECTORS 


A  Priori 

Probability  Vector 

Te»t 

Used  at  First 

Stage 

P1 

P2 

D 

*1 

L_ _ 

Y2 

Y3 

__  ■ 

EB 

Y1  3 
*  J 

v  i 

2,3 

m 

.5 

.5 

26.62 

27.14 

24.32 

34.59 

20.63 

.5 

.95 

14.16 

19.79 

11.06 

15.73 

11.62 

.9 

.7 

12.85 

*13 

13.76 

11.47 

13.61 

10.42 

12.27 

12.26 

.95 

.95 

4.07 

3.61 

5.78 

4.59 

i 

5.37 

4.10 

6.32 

5.74 

Table  11 

BEST  CORRECTIVE  ACTION  (a)  AS  A  FUNCTION  OF  THE  MESSAGE  RECEIVED  (y) 
FOR  SELECTED  FIRST-STAGE  TESTS  AND  PROBABILITY  VECTORS 


Test  Yj 

(P  -  (.95,  .95)) 

Test  Y 

(P  35  (. 

1,2 

5,  .95)) 

Test  Yl  3 
(P  -  (.9’,  .7)) 

Test  Yj 
(P  -  (.5 

,2,3 
,  .5)) 

yl 

yl.i 

ao 

yl,l 

ao 

yl,l,l 

a0 

y2,2,l 

*1,1,1 

y2 

ai.i 

yl,2 

*1,1 

yl >  1, 2 

*1,1,1 

y2,2,2 

*1,1,1 

y3 

a2 

y2,l 

*1,1 

y2,l 

*1,1 

yl  ,2, 1 

*1,1,1 

y3, 1 , 1 

*1,1.1 

y2,2 

y2 ,2 

*1.1 

yl  ,2,2 

*1,1,1 

y3, 1,2 

ai.i.i 

y3,l 

B9 

y3 , 1 

"2.1 

y2 , 1 , 1 

*1,1.1 

y3 , 2 , 1 

‘i.t.i 

_ 

y3,2 

Dfl 

y3 ,2 

a2.1 

y2,l,2 

*1,1,1 

y3 , 2 , 2 

‘1.1.1 
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other  possible  messages; 


p'(y, 

L  i 

,2' 

(.9, 

.7)) 

“  (.97, 

.18) 

^(.97, 

.18) 

=  25. 

49 

p'(y3 

*  i 

,1' 

(.9, 

.7)) 

-  (-82, 

.83) 

ft(.82. 

.83) 

«  11. 

56 

p'(y3 
^  ; 

X 

(.9, 

.7)) 

“  (-82, 

.18) 

V-82, 

.18) 

=  29. 

,05 

p'C/j 

X 

<•9, 

.7)) 

“  (0,  . 

83) 

fL(o, 

.83) 

-J 

p'<y3 

X 

(.9, 

.7)) 

=  (o,  . 

18) 

^(0, 

.18) 

-  J 

Step  2.  We  must  next  compute 


Fv  (y,  p)  2  rain  [c  (y,  a,  p)  +  f,(p'fy'(a.  p))) }. 

1,3  *1,3 


In  the  case  of  message  y^  for  example,  we  note  from  Table  9  that 


Yl,.Vy2,2 


5  ,  a 


O’  1,1*  1,2*  2 


i3- 

7  *" 


From  Table  7  we  obtain 


9  »  an  9  P) 

1,3  1 


cy  Cy?  a,  , ,  p)  “  ^  ^  *  i3, 

13  *  * 


CY  (y,  ,,  a  p)  =  4, 

1,3  ’ 


cv  (y,  9,  a  p)  »  6. 

1,3  *  *  ' 


^When  some  element  of  p'  Is  0,  f^J(p/)  is  automatically  set  equal 
to  infinity  in  order  to  rule  out  the  null  action.  See  Assumption  3. 
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Thus , 


?v  (y,  ,,  (.9,  .7)) 

*1,3  ’ 


*  Bln  [0  +  29.05,  10  +  6.93,  4  +  25.49,  6  +  11.36}  -  16. r" 

Consequently,  action  ^  is  optimal  when  message  y ^  ^  Is  re-*'.  .ed.  From 
the  formula  P(y)  =  P(y/z)P(z) ,  we  obtain,  by  reference  to  Tables  3  and  5 
P(y2  2)  *  (.2  x  .05) ( .9  x  .7)  +  (.2  x  .55)(.9  x  .3)  +  (.4  x  .05)(.l  x  .7)  + 
(.4  x  .55)(.l  x  .3)  •  .04.  Repeating  the  procedure  for  each  message,  we 
obtain: 


Fy  (y,  (.9,  .7)) 
1.3 

a*lt3,  (-9,  -7)(y) 

P(y) 

yl.l 

6.93 

t 

ao 

.59 

yl,2 

12.93 

•l.I 

' 

,15 

y2.1 

10.93 

!  *1.1 

.18 

y2,2 

16.93 

*1.1 

.04 

y3 , 1 

14.56 

a2,l 

.03 

y3,2 

20.56 

a2,l 

.01 

Note  that  the  first  and  third  columns  appear  in  Table  11. 

Step  3.  Since  T(Y^  ^ )  =  1-1  (Table  2),  we  obtain  from  the  preceding 
summary  that 


T(Y1  3}  +  Z  P(y)FY  (y’  (*9’  ,7))  =  1’1  +  9> 

*"1,3  1,3 


32  -  10.42. 


Since  this  is  the  minimum  value  given  by 


f  v  1 

min  |T(Y)  +  £  **ly)PY(y,  (.9,  .7 ))j, 


Y  CQJ 


y«Y 


f2(.9,  .7)  -  10.42. 


-42- 


IV.  THE  FINAL  CHECKOUT  PROBLEM  WHEN  TEST  RESULTS 
ARE  MULTIVALUED 

Id  the  preceding  model,  we  limited  the  discussion  to  the  case  in 
which  the  final  checkout  information  was  two-valued.  This  restriction, 
however,  has  not  been  present  with  respect  to  the  pre-checkout  tests  of 
parts,  since  the  model  is  the  same  whether  y  is  such  that  it  can  take 
only  two  values  or  whether  it  may  assume  several  values. 

Multivalued  test  results  correspond  to  the  situation  in  which  one 
learns  more  about  the  object  tested  than  merely  whether  it  passes  or 
falls  the  test.  Thus,  in  putting  a  system  through  a  final  test,  one  may 
often  ascertain  whether  specific  subsystems  work  even  though  the  system 
itself  fails  the  test. 

MULTIPLE  CHOICE  OF  TESTS 

Consider  a  system  of  K  components  in  which,  through  test  configura¬ 
tion  Y,  one  obtains  k  +  1  read-outs,  where  k  £  K/.  These  correspond  to 
k  +  1  messages,  one  representing  the  system  as  a  whole,  and  the  others 
k  different  components.  Let  the  indices  of  these  components  form  the 
set  Kj .  Thus,  we  may  write  y  =  (yQ,  y^,  y^,  ....  yfc) ,  where  each  y^  is 

two-valued  since  the  final  checkout  always  gives  perfect  information. 

k+ 1  k 

Thus  2  different  messages  may  be  distinguished  when  k  <  x',  and  2 

when  k  =  K*.  Let  S  be  such  that  the  components  about  which  specific 

information  is  received  are  at  the  beginning  of  S.  Letting  be  the 

cost  of  the  final  test  when  test  combination  Y  is  used,  the  cost  of  the 

final  checkout  (not  including  the  initial  system  test),  C^(S),  becomes 

when  this  sequence  is  used: 
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Cy(S)  -  nx0  +  (1  -  ppCRj  +  *[)  +  C*  +  ♦  (1  -  +  R,p 

+  Cfc  +  {1  -  p1,  ...  ,  pk)Ey  +  (1  -  nk+1>\+1  +  Ck+1 

*  Vi*r  +  -  +  -  V>V  +  V  *  Vs*  -  v*»'* 

where  the  last  t enn  is  zero  when  k  ^  K  .  Simplifying, 

(25)  CV(S)  -  l  (<*(■>*  ♦  »£>  ♦  0  ♦  U  -  P( . pk)^ 

keK1 

+  I  <»k  +  <  +  Vy> 


-  I 


Vk 


k^K, 


qK,EYK'  When  K1  ^ 

0  when  »  <p. 


After  test  Ey  Is  performed,  we  first  redesign  those  components  which 

the  test  shows  are  not  working.  With  the  remaining  components,  we 

proceed  as  in  Fig.  3  until  yQ  indicates  that  the  system  is  working. 

It  is  clearly  optimal  to  redesign  the  components  about  which  specific 

information  is  received  before  reworking  any  others.  To  see  this,  let 

component  j  6  Kj  be  transferred  to  the  position  following  i  i  K^.  This 

has  the  sole  effecc  of  changing  (1  -  tt  )B  to  (1  -  p  rr  )B  ,  m  ■  k  +■  1, 

a  m  j  m  m 

i,  and  (1  -  pp  ....  pk)Ey  to  (1  -  px,  ....  pjlpj+1,  pk)Ey 

+  (1  -  Pj)^,  all  of  which  increase  Cy.  This  gives  Theorem  6. 

Theorem  6.  Let  each  final  test  be  such  that  it  provides  Information 
as  to  whether  the  following  work:  ( 1)  The  system,  and  ;2)  each  component 
k  e  K, .  Then,  after  the  initial  system  test,  the  optimal  sequence  is  to 
redesign,  as  necessary,  all  components  k  8  Kj,  in  any  order.  At  this 


polrt,  proceed  as  in  Theorem  4  for  all  components  k  i  K^. 
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Thus  the  minimum  expected  cost  of  the  final  checkout  becomes, 
by  (25)  and  (10), 

(26)  f.(F)  -  olr.  ;e  +  C*  (F)  ). 

V'"l  1  1 

SERIAL  DEPENDENCE  AMONG  MESSAGES 

We  now  introduce  a  further  complication  with  respect  to  the  final 
checkout  of  a  system.  The  situation  to  be  discussed  is  one  in  which 
multivalued  messages  are  received  due  to  the  fact  that  subsystems  func¬ 
tion  serially.  That  is,  a  particular  subsystem  may  be  designed  to  go 
into  operation  only  after  other  subsystems  have  begun,  or  even  completed, 
their  functions.  Thus,  if  the  "lead"  subsystem  fails,  the  test  clearly 
cannot  piovide  information  with  respect  to  a  "successor"  subsystem.  We 
refer  to  "no  message"  as  the  null  message. 

An  example  of  this  kind  of  system  is  given  by  a  multistage  missile. 
Unless  the  first -stage  rocket  operates,  we  will  not  have  an  opportunity 
to  learn  whether  the  second  or  third  stages,  for  example,  work. 

Definition  1,  A  subsystem  belongs  to  the  first  stage  of  a  system 
operation^  if  its  activation  is  not  dependent  on  whether  some  other  sub¬ 
system  works. 

This  definition  is  readily  extended  by  induction  to 

Definition  2.  A  subsystem  belongs  to  the  nth  stage  of  a  system 
operation,  n  =  2,  3,  if  its  activation  is  dependent  on  whether 

a  subsystem  in  the  n  -  1th  stage  works  and  the  subsystem  does  not  already 
belong  to  a  previous  stage. 

The  various  3tages  of  system  operation  should  not  be  confused  with 
the  stages  (phases)  of  testing.  Unfortunately,  the  word  "stage"  is  the 
most  appropriate  in  both  instances. 
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Let  L  be  the  total  number  of  operational  stages  obtained  by  this 
procedure  and  let  K?  be  the  set  of  subsystem  indices  belonging  to  the 
jth  operational  stage,  j  =  1,  L,  for  which  p^  <  1.  Clearly,  L  £  K' , 

and  U._j  Kj  =  {1,  ....  K* } ;  that  is,  each  subsystem  is  identified  with 
one  and  only  one  operational  stage.  In  each  stage,  denote  by  i  K*  the 
set  of  subsystems  about  which  test  Y  provides  specific  information.  Now 
let  the  test  sequence  S  be  as  follows:  Let  all  subsystems  of  a  stage  be 
adjacent  to  each  other,  with  the  subsystems  about  which  specific  informa¬ 
tion  is  obtained  by  the  test  in  front  of  the  others.  Then  position  the 
operational  stages  in  ascending  order.  We  then  obtain 


L 

(27)  Cj(S)  -  £  [  l  +  *;>  +  c*)  +  (1  -  if  Pl)«, 

j«l  ieK.  ieK, 

J  J 

+  I  <B, +  c*  +  0it*  -  "ijV  • 

ukSk.  j  j 

j  j 

k' 

where  R. .  -  fi,  p,  ,  and  s.  is  the  last  subsystem  in  the  jth  stage, 
lj  k=l  k’  j 

Within  each  stage,  the  situation  is  clearly  completely  analogous  to  the 
one-stage  case  so  that  Theorem  6  applies  directly. 

Now  suppose  we  change  S  by  moving  subsystem  j  €  k/  to  the  position 
behind  subsystem  i  e  K^,  where  m  <  n.  Our  test  and  redesign  operations 
would  then  clearly  be  trapped  rn  the  rath  operational  stage  if  the  trans¬ 
ferred  subsystem  dees  not  work,  since  in  that  case  we  would  not  be  able 
to  obtain  any  information  about  subsequent  stages.  Since  >  0  for 
all  j  involved  by  Theorem  3,  there  is  a  positive  probability  that  this 


will  happen.  Thus  we  obtain 


-46- 


Thaorea  7.  Given  a  s/atea  that  ogertUi  in  L  stages  end  a  final 
teet  that  provides  information  ae  to  Aether  the  system  and  each  eub- 

it 

system  k  i  K  ,  J  *  1,  , . , ,  L,  work.  the  optimal  test  sequence,  S  ,  is 
obtained  as  follows:  {1)  Group  the  subsystems  by  stage  and  order  the 
stages  by  their  natural  number;  (2)  within  each  stage .  order  the  sub- 


systems  as  per  Theorem  6. 
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V.  PARALLEL  DEVELOPMENT  AS  A  SPECIAL  CASE 

An  interesting  attempt  to  model  the  phenomenon  of  parallel  RAD 
efforts  has  been  made  by  Nelson  [7].  We  now  demonstrate  how  that  model 
may  be  viewed  as  a  special  case  of  the  model  developed  in  this  study. 

Nelson  considers  the  problem  of  developing  s  system  to  meet  certain 
specifications  at  minimum  cost  in  time  and  money.  He  distinguishes 
between  two  stages:  design  and  development  completion.  In  the  design 
stage,  opportunities  for  learning  exist  that  may  be  beneficially  utilized 
In  the  second  stage.  These  opportunities  are  in  the  form  of  a  "built-in" 
test  which  gives  two-valued  perfect  Information  at  the  end  of  the  first 
stage  for  a  given  design. 

The  following  set  of  strategies  is  considered  by  Nelson:  The  devel¬ 
oper  has  the  opportunity  to  run  independent,  but  merit-wise  indistin¬ 
guishable,  projects  in  the  first  stage,  each  costing  the  same  amount  of 
money.  (As  before,  we  ignore  time  considerations.)  Then,  at  the  end 
of  the  first  stage,  he  is  to  pick  one  project  to  run  to  completion.  We 
denote  a  success  in  the  first-stage  teat  of  a  design  by  y^  and  failure  by 
y^.  For  any  number  of  designs  is  there  is  clearly  no  benefit  in  obtaining 
more  than  one  y^  since  only  one  design  is  to  be  pursued  further.  Thus 
there  are  only  two  macrotnessages  that  are  relevant  for  a  given  i;  let  y^j 
be  the  set  of  messages  containing  at  least  one  y^  among  the  i  designs 
tested  and  let  y^  be  the  message  that  y  =  y^  for  all  i.  Let  us  also  make 
the  following  definition#: 

M  55  average  cost  of  a  design  incurred  in  the  first  stage,  including 
testing 

p  “  probability  of  obtaining  message  y^  for  a  design 
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s  total  test  and  redesign  cost  In  second  stage  when  y^j  ts 
obtained 

*  total  test  and  redesign  cost  in  second  stage  when  y ^  is 
obtained 

K  *  total  development  cost  in  second  stage  exclusive  of  testing 
and  redesign 

When  i  designs  are  pursued  in  the  first  stage,  the  total  expected 
development  cost  of  the  system  becomes 

(28)  M  +  (i  -  1)M  +  K  +  (1  -  pSm^  +  p1^. 

Thus,  since  M  and  K  are  fixed,  the  minimum  expected  cost  of  testing 
and  redesign  is 

(29)  min  {(i  -  1)M  +  (1  -  p^M.  +  pSl  } , 

iel  *  * 

where  I  is  the  set  of  positive  Integers. 

Comparing  (29)  with  (22),  we  find  that 


V  =  I, 

Yi  =  l* 

T(Yi)  -  (i  -  1)M, 

p(yn>  =  i  -  p1, 

p(y12)  =  p1, 

*  — 

=Y(y»  a,  p)  =  0, 

y/  ■  >'11’ 
c  (p'(p,  y i j > )  *  Mj . 
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ao  that  the  preceding  model  of  parallel  development  may  be  viewed  ac 
a  special  case  of  the  testing  problem  considered  in  this  study* 
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